Skip to content

Pods: Gang Scheduling & Topology awareness #494

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
Varsius wants to merge 10 commits into
base: main
Choose a base branch
from
Draft

Pods: Gang Scheduling & Topology awareness #494

Varsius wants to merge 10 commits into from

Conversation

@Varsius
Copy link
Collaborator

@Varsius Varsius commented Feb 4, 2026 •
edited
Loading

Overview

This PR introduces Gang Scheduling (#393) in conjunction with Topology Aware Scheduling (TAS) (423) to the pods scheduler, as well as a refactoring of the overall scheduling behavior (includes #421).

TODO

The code is in a working state, however the PR is kept in draft state since there is still work needed in order to get a clean integration into the project:

  • Add back the PodGroupSetController to handle the creation of pods from scheduled PGSs which is currently done by the scheduler
  • Add unit tests for all newly added and refactored components:
    • Scheduling Cache
    • Event Handlers
    • PodGroupSet Scheduling
    • Pod Scheduling
    • Scheduling Queue
    • Scheduler
    • Topology
  • List and implement the various TODO comments throughout the code
  • Documentation for PodGroupSets and Topology labels

Changelog

  • Introduce PodGroupSet (PGS) as a new CRD which is used for gang scheduled workloads
  • Introduce a scheduling cache for tracking the state of the cluster's nodes and topology
    • Events are received using informers and state of the pod/PGS are fetched from the respective lister
    • Add code generation to get a client, informer and lister for PGSs
  • Introduce a priority scheduling queue similar to the kube-scheduler
    • ActiveQueue: workloads that should be scheduled immeadietly
    • BackoffQueue: workloads that could not be scheduled due to temporary failures, e.g. pipeline not ready
    • UnschedulableQueue: workloads that are waiting for some event, e.g. new node or other pod completes
    • Priority assignment using Largest-Gang-First-Served (LGFS)
  • Reduce responsibilities of PipelineDecisionController to reconciling pipeline steps
  • Introduce PodGroupSetController that handles the lifecycle events of PodGroupSets
  • Introduce scheduling loop as a separate go-rountine
    • Replaces the reconciliation of decisions for scheduling pods
    • Enables the unified scheduling of both pods and PGSs
    • Enables usage of the scheduling queue for event based scheduling retry attempts
  • Introduce Topology data structure for hierarchical scheduling attempts for PGSs which is used for TAS
    • The topology is defined by using node labels, e.g. cortex/topology-rack=rack-23

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@auhlig auhlig Awaiting requested review from auhlig auhlig will be requested when the pull request is marked ready for review auhlig is a code owner

@juliusclausnitzer juliusclausnitzer Awaiting requested review from juliusclausnitzer juliusclausnitzer will be requested when the pull request is marked ready for review juliusclausnitzer is a code owner

@umswmayj umswmayj Awaiting requested review from umswmayj umswmayj will be requested when the pull request is marked ready for review umswmayj is a code owner

@mblos mblos Awaiting requested review from mblos mblos will be requested when the pull request is marked ready for review mblos is a code owner

@SoWieMarkus SoWieMarkus Awaiting requested review from SoWieMarkus SoWieMarkus will be requested when the pull request is marked ready for review SoWieMarkus is a code owner

@PhilippMatthes PhilippMatthes Awaiting requested review from PhilippMatthes PhilippMatthes will be requested when the pull request is marked ready for review PhilippMatthes is a code owner

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@Varsius